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Do the Most Cited Web Sources 
Provide Proper Information on Influenza 
Vaccination? 
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ABSTRACT 


Objective: The Internet is increasingly preferred as a source of health-related information. 
However, several studies revealed deficiencies in the quality of web-based health 
information. Therefore, this study aimed to analyze the content and design of influenza 
vaccine-related websites. 


Materials and Methods: This study was conducted using a search engine, Google Search, in 
the Turkish language. We identified 15 influenza-related search terms, and for each term, 
the top ten links on the first page were included in the study after removing duplicate links. 
Then, we evaluated identified websites in terms of content and design criteria. 


Results: Seventy-one links related to influenza vaccination were evaluated in our study. 
The mean content criteria score of all selected websites was 11.5 points. While academic 
websites received the highest contents point with 22.5, news and hospital sites remained 
under the average with 10.5 and 8.9 points, respectively. Thirty-eight per cent of the 
websites had aesthetic/content professionalism, and 77.5% included contact information. 
References were shown only in 14.1% of websites. The websites with higher design criteria 
did not have higher content points. 


Conclusion: The Internet remains a frequently referenced source of information. The 
accuracy of the information on health topics included on the Internet is important in terms 
of public health. Unfortunately, according to our study, the top websites giving information 
on influenza vaccination were not well designed and had insufficient content. 
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Web Sources for Influenza Vaccination 


INTRODUCTION 


nfluenza constitutes one of the major threats 

to the health system, affecting up to 10% of the 

world’s population every year (1, 2). According 
to the Centers for Disease Control and Preven- 
tion (CDC) database, between 1999-2015, 291,000- 
646,000 deaths caused by influenza-associated 
respiratory disease were recorded worldwide (3). 
Seasonal influenza viruses can cause high morbid- 
ity and mortality, especially in people with under- 
ying diseases, the elderly, pregnant women and 
children (4). Influenza vaccination is the primary 
prevention method of influenza infection (5). 


Local policies, knowledge and perceptions can in- 
fluence the vaccination rate in different communi- 
ties. The rate of influenza vaccination and factors 
affecting infection control team (ICT) decision on 
getting vaccinated was assessed in a multination- 
al study. In Turkey, the overall vaccination rate in 
the 2018-2019 season was only 39%, even among 
ICT members primarily responsible for the vacci- 
nation of health care professionals. Not being in a 
risk group and not believing the vaccine’s efficacy 
were the major reasons for not getting vaccinated 
(6). Therefore, it is important to find out the reasons 
affecting vaccination decision and provide reliable 
vaccine-related information to enhance the vacci- 
nation rates. 


The Internet remains a frequently referenced 
source of information, and the Internet usage rate 
has been augmented in Turkey as well as all over 
the world (7). According to the Turkish Statistics In- 
stitute Information and Communication Technolo- 
gy (ICT) Usage Survey on Households and Individu- 
als, 2019, the Internet access rate throughout Tur- 
key increased to 94.9% in 2019 (8). The Internet is 
increasingly preferred as a source of health-related 
information (9). However, several studies revealed 
deficiencies in the quality of web-based health in- 
formation (10). For example, some of the news on 
the websites can have positive/neutral/negative at- 
titudes towards influenza vaccination (9). Despite 
the wealth of evidence that vaccines prevent dis- 
ease and save millions of lives every year, vaccine 
hesitancy became an important challenge in pub- 
lic health (11). It causes delay, and vaccine refus- 
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al becomes a risk for vaccine-preventable disease 
outbreaks (11). Also, social media has allowed am- 
plifying and spreading of antivaccination messages 
(12). Therefore, the contents and formats of web- 
sites giving vaccine information must be planned 
and tested carefully. Our study aimed to analyze 
the content and design of influenza vaccine-related 
websites. 


MATERIALS AND METHODS 


Ankara University Faculty of Medicine Ethical 
Committee approved the study. This cross-sec- 
tional study was conducted using the most com- 
mon search engine, Google Search, available in 
the Turkish language. The study was conducted 
between November 1, 2016, and December 1, 2016. 
We searched for 15 search terms related to influ- 
enza (H1N1, influenza vaccine, vaccine mandate, 
“who should be vaccinated”, “who should not be 
vaccinated”, elderly, childhood, malignancy, preg- 
nancy, ingredients of the vaccine, vaccination at 
the pharmacy, the timing of vaccination, benefits of 
vaccination, harms of vaccine, vaccine marketing). 
The top ten links on the first page were included 
for each search term, and duplicated links were ex- 
cluded from the study. The selected websites were 
evaluated according to a list of content and design 
criteria. Contents of websites on influenza vaccine 
were assessed according to key facts about influen- 
za vaccination determined by CDC in 2016 (13). The 
information on the websites was graded from 1 to 
3 according to the medical content (1: general in- 
formation; 2: detailed information; 3: very detailed/ 
elaborated information) (Table 1). Vaccine Safe- 
ty Net (VSN), The Global Advisory Committee on 
Vaccine Safety (GACVS) initiative, developed four 


HIGHLIGHTS 


e The Internet, which is increasingly preferred as a 
source of health-related information, may have 
deficiencies in the quality of information. 


¢ Common network resources may not contain 
sufficient vaccine-related information. 

e The contents and design of websites giving vaccine 
information must be planned and tested carefully. 
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categories of criteria (credibility, content, accessi- 
bility and design) in order to assess information on 
vaccines publicly available on the Internet. World 
Health Organization (WHO) design standardization 
based on VSN criteria was used to detect the suitabil- 
ity of websites design (14). Presence of references, up- 
to-dateness, having vision/mission, presence of con- 
tact information, aesthetic/content professionalism, 
accessibility of websites were assessed based on this 
set of criteria. Sites were encoded as 1 or 0 depending 
on whether they met the evaluation criteria. 


Statistical analyses were performed by using Sta- 
tistical Package for the Social Sciences (SPSS) soft- 
ware version 18.0. The results of descriptive statis- 
tics were reported as mean + standard deviation 
for variables with normal distribution. Categorical 
data were reported as frequencies and percentages. 
The chi-square test was used to assess the signifi- 
cant association between categorical variables. Sta- 
tistical significance was set at p < 0.05. 


RESULTS 


Totally 150 links were obtained, 71 were evaluated 
after excluding duplication. Websites were catego- 
rized into academic, news, personal blog, hospital, 
portal sites and others (Figure 1). We applied 15 con- 
tent criteria to determine the property of websites 
content (Table 1). The mean content criteria score 
of all selected websites was 11.5. The content points 
for each category are shown in Figure 2. While ac- 
ademic websites received the highest content point 
with 22.5, news and hospital sites remained below 
average with 10.5 and 8.9 points, respectively. Less 
than fifty percent of all websites met criteria other 
than criteria 2, 4, and 6. Criteria 2 (vaccine should be 
repeated every year), 4 (risk groups should be vacci- 
nated) and 6 (the recommended months for vaccina- 
tion) were met by 53.5%, 74.7%, 59.2% of all websites, 
respectively. Criteria 1 (2.8%), 10 (5.6%), 11 (4.2%) 
were satisfied with the lowest rates in the websites. 


Websites were evaluated according to WHO design 
criteria. Thirty-eight percent and 77.5% included 
aesthetic/content professionalism and contact 
information, respectively. References were found 
only in 14.1% of websites (Figure 3). Websites 
whose design criteria scores were 1, 2, 3 and 4 had 
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Table 1. The significance level of content criteria of the website about influenza vaccination. 


12, 11.4, 10.4 and 14 points for content criteria, re- 
spectively. The websites with higher design crite- 
ria did not have higher content points. 


DISCUSSION 


Influenza is an important viral respiratory disease 
with a significant burden on public health. There- 
fore, many interventions focus on the prevention 
of influenza. Preventive measures include vacci- 
nation, hand hygiene and surgical masks (15). The 
best method for controlling and preventing influen- 
za disease is vaccination, reducing the incidence of 
the disease and its severity (16). 


Vaccination is considered a milestone in public 
health because it significantly affects morbidity 
and mortality rates of infectious diseases (17). A 
wide range of factors can potentially influence the 
vaccination decision (18). Over the past few de- 
cades, vaccine hesitancy has become a significant 
threat to public health, resulting in outbreaks such 
as measles (19). The misinformation propagated by 
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Number Content criterions Point 
d Higher dose influenza vaccination for people 65 years and older 1 
2 Repeating influenza vaccination every year because of changing epidemiology 3 
Administration of vaccine intramuscularly 2 
4 Risk groups for influenza 3 
2 Who should not be vaccinated 2 
6 The recommended months for vaccination 3 
Where can people get an influenza vaccine 2 
8 Protection of vaccine starts two weeks after administration 2 
9 People can get seasonal influenza even though a vaccine this year 2 
10 nfluenza vaccination is required even after getting flu 1 
11 Cross protection of influenza vaccine 1 
12 Determination of the strains found in that season’s vaccine 1 
13 Clinical course and complications of influenza 3 
14 Local side effects of influenza vaccine 2 
15 Systemic side effects of influenza vaccine 2 
Total 30 


anti-vaccine groups and media augmented con- 
cerns about vaccines (20). Although many scien- 
tific data support the safety of the recommended 
vaccines, fighting against false information to per- 
suade people with vaccine hesitation continues to 
be a major problem (19). 


Most people search for health information online 
especially using search engines (21). However, there 
are concerns regarding the accuracy and reliability 
of the Internet besides it provides immediate and 
easily available information (22). Misleading and 
over-interpreted health news in social media can be 
a potential threat to public health. A study inves- 
tigating the spread of fake medical news showed 
that the most misleading content was about vac- 
cines (23). Anti-vaccine websites generally focus 
on the ineffectiveness and danger of vaccines with 
incorrect interpretations of scientific reports (21). 
People’s knowledge and attitude about influenza 
vaccination can be affected by the output of search 
engines (15). Inappropriate information is not only 
found on websites but also on other media plat- 
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forms (24). For example, Frazier et al. showed that 
inaccurate information about hand hygiene was 
provided 76% from television, 53% from magazines, 
12% from newspapers (25). 


Seventy percent of studies concluded that the qual- 
ity of health information on websites was an im- 
portant problem (24). Significant differences can be 
observed between websites categories in terms of 
content (22). For example, scientific websites pro- 
vided more appropriate information for idiopathic 
pulmonary fibrosis than news and personal blog 
websites (22). Also, in our study, we showed that 
academic websites’ content points (22,5 points) 
were higher than news (10.5 points) and personal 
blogs (11.5 points). The scores of news websites, 
which constitute a significant part of the Internet, 
were below the average. There was not any website 
with total points based on content criteria. 


The websites with higher WHO design criteria did 
not have higher content points. It was seen that 
websites meeting the WHO design criteria did not 
have appropriate content. Therefore, we conceived 
that the WHO design criteria did not warrant the 
appropriate information. 


Our study had some limitations. This research evalu- 
ated the content and design of vaccine-related web- 
sites in the Turkish language. Our results can differ 
from the average user's results because Google Search 
uses the user's location to personalize the results. An- 
other limitation was the dynamic nature of the web. 
We limited our search to Turkish-language websites so 
that the results would differ in other languages. 
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Websites of professional organizations such as The 
European Society for Clinical Microbiology and In- 
fectious Disease (ESCMID), Infectious Diseases So- 
ciety of America (IDSA), Turkish Clinical Microbi- 
ology and Infectious Disease Society (KLIMIK) are 
some of the most trustworthy information sources 
about this subject. However, the websites of these 
societies were not found among the first ten links 
during our search, and we could not evaluate their 
websites regarding design and content. Therefore, 
the reliability of these sources requires further in- 
vestigation. 


The Internet remains a largely unregulated enti- 
ty, so it is important to follow up on the impact of 
health-related information on public health (15). 
Several companies have developed different crite- 
ria lists for guiding and evaluating health-related 
websites’ content, but these criteria have not been 
applied routinely. Instead, these systems rely on 
voluntary self-assessment by web page developers, 
the acceptability and reliability of most of them is 
unknown (10). 


In conclusion, despite the varying quality, the In- 
ternet will remain a preferred resource for patients 
given the easy access to health-related topics. The 
ongoing use of the Internet necessitates a strate- 
gy to increase the accuracy of online information, 
highlighting and facilitating access to websites with 
appropriate content (22). We demonstrated that 
most cited network resources did not contain ap- 
propriate and satisfactory information about influ- 
enza vaccination. 
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